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Determining  the  Best  Loci  of  Knowledge,  Responsibilities  and 
Decision  Rights  in  Major  Acquisition  Organizations 

Presenter:  John  Dillard,  joined  the  NPS  faculty  in  the  fall  of  2000  with  extensive 
experience  in  the  field  of  systems  acquisition  management.  His  research  focuses  on  defense 
acquisition  policy  changes  and  their  implications.  John  began  his  career  in  program  and 
contract  management  after  attaining  a  MS  in  Systems  Management  from  the  University  of 
Southern  California  in  1985.  He  has  been  involved  with  myriad  technologies  and  system 
concepts  that  have  evolved  into  fielded  products,  such  as  the  M-4  Carbine,  120mm  Mortar,  and 
M-24  Sniper  Weapon.  He  was  the  Assistant  Project  Manager  for  Development  of  both  the  Army 
Tactical  Missile  System  and,  later,  the  JAVELIN  Antitank  Weapon  System  at  Redstone  Arsenal, 
Alabama.  All  of  these  systems  incorporate  state-of-the-art  technologies,  are  in  sustained 
production  and  fielding,  and  are  now  battle-proven.  He  was  the  Product  Manager  for  the  Joint 
Advanced  Special  Operations  Radio  System,  and  in  1998  was  appointed  to  head  Defense 
Department  contract  administration  in  the  New  York  metropolitan  area.  John  has  consulted  for 
the  governments  of  Mexico  and  the  Czech  Republic  on  achieving  excellence  in  the  public 
sector.  As  an  adjunct  professor  for  the  University  of  California  at  Santa  Cruz,  he  teaches 
courses  in  project  management  and  leadership  to  Silicon  Valley  public-  and  private-industry 
professionals. 

John  Dillard 
Senior  Lecturer 
Naval  Postgraduate  School 
Monterey,  CA  93943-5197 
Phone:  (831)  656-2650 
Fax:  (831)  656-3407 
E-mail:  jdillard@nps.edu 

Presenter:  Mark  Nissen,  is  Associate  Professor  of  Information  Systems  and 
Management  at  the  Naval  Postgraduate  School.  His  research  focuses  on  knowledge  dynamics. 
He  views  work,  technology  and  organizations  as  an  integrated  design  problem  and  has 
concentrated  recently  on  the  phenomenology  of  knowledge  flows.  Mark’s  publications  span 
information  systems,  project  management,  organization  studies,  knowledge  management  and 
related  fields.  In  2000,  he  received  the  Menneken  Faculty  Award  for  Excellence  in  Scientific 
Research,  the  top  research  award  available  to  faculty  at  the  Naval  Postgraduate  School.  In 
2001,  he  received  a  prestigious  Young  Investigator  Grant  Award  from  the  Office  of  Naval 
Research  for  work  on  knowledge-flow  theory.  In  2002,  he  spent  his  sabbatical  year  at  Stanford 
integrating  knowledge-flow  theory  into  agent-based  tools  for  computational  modeling.  Before  his 
information  systems  doctoral  work  at  the  University  of  Southern  California,  he  acquired  over  a 
dozen  years'  management  experience  in  the  aerospace  and  electronics  industries. 
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Associate  Professor 
Naval  Postgraduate  School 
Monterey,  CA  93943-5197 
Phone:  (831)  656-3570 
Fax:  (831)  656-3407 
E-mail:  mnissen@nps.edu 
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Abstract 

The  DoD  is  a  large,  bureaucratic,  rule-intensive  organization  that  may  not  be  suited  well 
for  its  environment.  Building  upon  prior  research  of  acquisition  centralization  and  knowledge 
dynamics,  we  employ  computational  methods  to  assess  the  behavior  and  performance  of 
different  organizational  designs  in  varying  environments.  Our  results  reinforce  Contingency 
Theory  and  suggest  particular  characteristics  of  different  acquisition  environments  make  one 
organizational  form  relatively  more  or  less  appropriate  than  another.  Practically,  answers  to  our 
research  questions  have  direct  and  immediate  application  to  acquisition  leaders  and  policy 
makers.  Theoretically,  we  generalize  to  broad  classes  of  organizations  and  prescribe  a  novel 
set  of  organizational  design  guides. 

Introduction 

Acquisition  is  big  business.  The  US  Department  of  Defense  (DoD)  alone  executes 
routinely  eleven-figure  budgets  for  research,  development,  procurement  and  support  of  weapon 
systems,  for  instance.  Acquisition  is  also  a  rule-intensive  business.  In  addition  to  myriad  laws 
governing  federal  acquisition  in  the  US,  a  plethora  of  regulations  specify — in  great  detail  often — 
how  to  accomplish  the  planning,  review,  execution  and  oversight  of  Government  acquisition 
programs,  large  and  small,  sole-source  and  competitive,  military  and  commercial  (Dillard,  2003). 
Due  in  great  part  to  the  large  size  and  many  rules  associated  with  Defense  acquisition  in 
particular,  the  organizations  responsible  for  DoD  acquisition  activities  tend  to  be  large  and  rule¬ 
intensive  themselves,  reflecting  the  kinds  of  centralized,  formalized,  specialized  and  oversight¬ 
intensive  forms  corresponding  to  the  classic  Machine  Bureaucracy  from  Organization  Theory 
(e.g.,  see  Mintzberg,  1979).  The  problem  is  this  classic  organizational  structure  is  known  well  to 
be  exceptionally  poor  at  responding  to  change.  In  the  context  of  military  transformation,  such  a 
problem  should  be  clear  and  compelling.  Arguably,  one  or  more,  superior,  organizational 
approaches  must  be  available  to  replace  the  current  acquisition  organization.  But  which,  if  any, 
is  most  appropriate?  On  what  basis  should  acquisition  leaders  and  policy  makers  choose 
between  such  competing  organizational  forms?  What  evidence  supports  claims  of  superiority  for 
one  organizational  approach  versus  another?  Questions  such  as  these  are  difficult  to  answer 
through  most  research  methods  employed  today  to  study  acquisition  organizations  (e.g.,  case 
study,  survey,  action). 

The  bureaucratic  nature  of  the  DoD  Acquisition  Organization  did  not  emerge  recently, 
nor  did  it  materialize  by  design.  Rather,  it  reflects  the  cumulative  accretion  of  laws,  regulations, 
rules  and  hierarchical  levels  over  considerable  time.  If  only  the  organization  could  be  changed 
and  evaluated — say  through  assessment  of  four  alternate  organizational  structures — then  one 
could  assess  the  relative  performance  of  the  new  organizational  designs  versus  the  current  form 
and  recommend  transformation  toward  the  best  performer.  But,  clearly  the  set  of  problems  and 
actors  in  the  changed  organizations  would  differ  from  those  associated  with  the  original  and  with 
one  another;  that  is,  there  is  no  way  to  impose  controls  over  such  a  study  (e.g.,  internal  validity 
is  compromised).  This  is  one  reason  why  so  many  acquisition  research  projects  produce  so  little 
new  knowledge.  Alternatively,  such  controls  can  be  imposed  easily  through  laboratory 
experimentation.  Yet,  the  simplified  nature  and  laboratory  context  of  experiments  fail  to  capture 
the  size,  scope  and  complexity  of  the  acquisition  organization  (e.g.,  external  validity  is 
compromised).  This  is  another  reason  why  so  many  acquisition  research  projects  produce  so 
little  new  knowledge.  However,  by  combining  the  best  features  of  laboratory  experimentation 
(e.g.,  experimental  controls)  with  field  methods  (e.g.,  large-scale  and  complex  behaviors),  one 
can  design  and  conduct  a  study  of  acquisition  organizations  that  reflects  both  internal  and 
external  validity. 
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This  is  the  approach  of  computational  experimentation:  using  sophisticated  and 
validated  computer  models  of  organizations  to  assess  the  behavior  and  performance  of  different 
organizational  designs.  Computational  Organization  Theory  (COT;  see  Carley  &  Prietula,  1994) 
provides  a  set  of  methods  and  tools  to  enable  this  approach.  In  particular,  using  the  methods 
and  tools  associated  with  the  Virtual  Design  Team  (VDT)  Research  Group  at  Stanford, 
computational  models  of  organizations  are  driven  by  well-accepted  organization  theory  and  are 
validated  by  extensive  and  repeated  field  studies.  This  validation  provides  considerable 
confidence  that  computational  results  reflect  the  likely  behaviors  and  performance  of  the 
acquisition  organizations  they  model  and  emulate. 

The  research  described  in  this  article  involves  the  application  of  VDT  methods  and  tools 
to  study  acquisition  organizations.  In  particular,  we  model  and  simulate  the  behavior  of 
organizations  associated  with  major  defense  acquisition  programs  in  the  DoD.  We  provide  both 
answers  to  and  insights  into  how  such  organizations  can  be  changed  to  improve  performance. 
Some  of  the  key  organizational  design  variables  of  interest  pertain  to  the  bureaucratic  nature  of 
the  organization  and  follow  from  recent  research  to  investigate  centralization  (Dillard,  2003).  For 
instance,  factors  such  as  centralization,  formalization,  specialization,  hierarchical  layers  and  the 
like  can  be  manipulated — individually  as  well  as  in  combination — under  controlled  and  replicated 
conditions  to  assess  the  performance  of  acquisition  organizations  in  different  forms.  This  follows 
recent,  complementary  research  using  computational  organization  theory  in  the  domain  of 
military  command  and  control  (Nissen  &  Buettner,  2004;  Nissen,  2005a,  b).  Considerations  such 
as  the  number,  frequency  and  level  of  acquisition  reviews,  adaptability  and  flexibility  of 
acquisition  organizations,  and  risk-versus-project-duration  of  acquisition  programs  are  primary 
in  this  study.  The  key  research  question  is:  How  can  organizations  responsible  for  major 
acquisition  programs  be  redesigned  to  improve  performance? 

The  significance  of  this  approach  is  twofold.  First,  answers  to  the  research  question 
have  direct  and  immediate  application  to  acquisition  leaders  and  policy  makers.  Such  answers 
address  a  serious  and  immediate  problem,  revealing  insights  into  the  behaviors  of  major 
acquisition  organizations  that  are  too  complex  and  dynamic  to  be  understood  well  or  directly. 
They  illuminate  the  kinds  of  changes  acquisition  organizations  can  make  to  balance  competing 
performance  measures  (e.g.,  adaptability  &  flexibility  vs.  project  risk  &  duration).  They  can 
explain — in  a  theoretically  grounded  manner — many  different  cases  of  acquisition  success  as 
well  as  failure.  They  can  also  provide  overarching  theory  to  help  promote  the  former  and  obviate 
the  latter  in  future  acquisition  programs. 

Second,  this  research  project  demonstrates  the  efficacy  of  a  new  approach  to  studying 
acquisition  organizations.  It  enables  leaders,  policy  makers  and  analysts  to  answer  “how  much” 
questions  such  as:  How  much  centralization,  formalization  and  specialization  is  best?  What 
fraction  of  commercial  off-the-shelf  equipment  would  be  ideal?  What  level  of  concurrency 
between  development  and  production  provides  the  best  combination  of  cost,  schedule, 
performance  and  risk?  Such  questions  are  not  answered  well  today  in  terms  of  acquisition 
organizations.  This  leaves  acquisition  decision  makers  today  with  no  reliable  means  to  address 
such  questions. 

The  balance  of  the  article  begins  with  a  focused  review  of  the  literature  relevant  to  this 
study.  We  follow  with  discussion  of  our  research  design  and  description  of  the  computation 
model  developed  to  represent  and  emulate  the  acquisition  organization.  The  article  turns  then  to 
discuss  results  of  our  computational  experiments.  Conclusions,  implications  and 
recommendations  for  future  research  close  the  article,  along  with  a  rich  set  of  references  for 
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deeper  exploration  into  the  research  on  which  this  article  builds  and  contributes.  We  also 
include  two  appendices  to  provide  details  of  our  computational  models. 

BACKGROUND 

This  focused  review  of  the  literature  relevant  to  our  study  is  organized  into  three  parts:  1) 
the  acquisition  organization,  2)  organization  theory,  and  3)  computational  experimentation. 


The  Acquisition  Organization 

Of  particular  interest  to  the  authors  is  the  realm  of  DoD  program  management,  where 
research  and  development  dollars  are  expended  to  invent  or  advance  warfighting  capability. 
While  US  weaponry  is  considered  some  of  the  best  in  the  world,  the  major  acquisition  projects 
to  acquire  them  are  often  fraught  with  cost  and  schedule  growth.  They  even  fail  at  times  to  meet 
specifications  or  to  provide  the  capabilities  desired.  Since  implementation  of  the  Goldwater- 
Nichols  Act  legislation  in  the  late  1980s,  major  defense  acquisition  organizations  (e.g.,  program 
management  offices)  have  operated  under  a  four-tiered  decision  structure. 

For  major  acquisitions,  the  current  policy  makes  clear  that  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics  is  the  Milestone  Decision  Authority 
responsible  for  the  overall  program:  Described  in  the  DODI  5000.1 : 

3.4  The  Milestone  Decision  Authority  (MDA)  is  the  designated  individual  with  overall 
responsibility  for  a  program.  The  MDA  shall  have  the  authority  to  approve  entry  of  an 
acquisition  program  into  the  next  phase  of  the  acquisition  process  and  shall  be 
accountable  for  cost,  schedule,  and  performance  reporting  to  higher  authority,  including 
Congressional  reporting.  (USD(AT&L),  2003) 

And  three  levels  down  the  hierarchy,  Program  Managers  (PMs)  are  described  as: 

3.5.1  the  designated  individual  with  responsibility  for  and  authority  to  accomplish 
program  objectives  for  development,  production,  and  sustainment  to  meet  the  user's 
operational  needs.  The  PM  shall  be  accountable  for  credible  cost,  schedule,  and 
performance  reporting  to  the  MDA.  (USD(AT&L),  2003) 

Thus,  the  Program  Manager  and  Milestone  Decision  Authority  share  responsibility  for 
development  and  oversight  of  a  program.  Further  guidance  under  the  DoD  Instruction  5000.1 
provides: 

4. 3. 1.1  There  is  no  one  best  way  to  structure  an  acquisition  program  to  accomplish  the 
objective  of  the  Defense  Acquisition  System.  MDAs  and  PMs  shall  tailor  program 
strategies  and  oversight,  including  documentation  of  program  information,  acquisition 
phases,  the  timing  and  scope  of  decision  reviews  and  decision  levels  to  fit  the  particular 
conditions  of  that  program,  consistent  with  applicable  laws  and  regulations  and  the  time- 
sensitivity  of  the  capability  need.  (USD(AT&L),  2003) 

However,  while  the  wording  above  might  indicate  that  the  MDA  and  PM  plan  jointly  or 
collaborate  in  some  way  on  program  strategy,  there  are,  in  fact,  both  a  Component  Acquisition 
Executive  and  Program  Executive  Officer  in  the  hierarchy  between  them,  and  direct 
communication  between  MDA  and  PM  is  infrequent.  The  four  tiers  of  major  program  command 
and  control  and  typical  grade/ranks  of  positions  are  shown  in  Figure  1 . 
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Defense  Acquisition  Board  Chairman 
Political  Appointee  -  SES 
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Figure  1.  DoD  Decision  Hierarchy  for  Major  Defense  Acquisition  Programs 

(adapted  from  DAU,  2004) 

MDA  PMs  lead  Program  Management  Offices  (PMOs).  PMOs  vary  greatly  in  size.  A 
typical  range  of  government-assigned  workers  is  generally  between  50  and  100  individuals 
dedicated  to  the  day-to-day  efforts.  An  expanding  network  of  other  government  agency  players, 
multi-tier  industry  contractors,  and  other  participants  can  multiply  this  figure  many  times  (Dillard, 
2004).  While  all  stakeholders  represent  different  parts  of  the  enterprise,  here  we  refer  to  this 
central  government  organizational  entity — the  government  PMO — as  the  acquisition 
organization. 

At  the  PMO  level,  several  alternatives  for  the  organization  exist.  In  most  cases,  the 
offices  are  comprised  of  permanently  assigned  “core”  personnel,  and  temporarily  assigned  co¬ 
located  “matrix”  personnel  on  loan  from  commodity  systems  commands.  These  are  personnel 
typically  arrayed  by  functional  area  within  the  PMO  (as  shown  in  Figure  2).  A  significant  number 
of  on-site  support  contract  personnel  may  be  present  as  well. 
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PROGRAM  OFFICE  ORGANIZATION  STRUCTURE 


Figure  2.  Typical,  Matrixed  Program  Management  Office  Structure  (adapted  from  DAU,  2004) 

Somewhat  less  formally,  programs  also  organize  internally  in  ad  hoc  teams  oriented  on 
specific  areas  of  each  project.  This  stems  largely  from  DoD  initiatives  over  the  last  10  years  to 
implement  Integrated  Product  and  Process  Development  (IPPD)  using  Integrated  Product 
Teams  (IPT).  This  management  philosophy  emphasizes  the  potential  of  collective  knowledge 
via  small  organizations  with  cross-functional  or  multi-disciplinary  members  (OUSD,  1998). 
Interestingly,  the  ideas  in  this  IPPD/IPT  philosophy  of  work  implementation  and  problem  solving 
are  also  embodied  and  magnified  in  emerging  thought  regarding  command  and  control  (C2)  in 
tactical  military  organizations.  The  text  Power  to  the  Edge  recognizes  the  benefit  of  using 
information-age  technology  to  transfer  knowledge  and  power  to  the  point  of  an  organization’s 
interaction  with  its  environment  (Alberts  &  Hayes,  2003). 


INTEGRATED  PRODUCT  TEAMS 


IPT  =  Integrated  Product  Team 
PIT  =  Program  Integration  Team 

Figure  3.  Example  (Aircraft)  PMO  IPT  Structure  (adapted  from  DAU,  2004) 
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Another  concept  pertinent  to  our  introduction  is  that  of  work  and  organizational 
hierarchy.  Nobel  Prize  winner  Herbert  Simon  argues,  from  his  observation  of  complexity  in 
things  both  natural  and  artificial,  that  complex  systems  evolve  from  simple  systems.  And  they  do 
so  more  rapidly  when  there  are  stable,  intermediate  forms  or  sub-systems  (like  modules  or 
“units  of  action”).  Moreover,  he  argues  the  resulting  evolution  into  the  complex  system  will  be 
hierarchic,  including  systems  such  as  organizations  (Simon,  1981).  But  an  important 
observation  is  also  made  by  Koestler,  who  studies  hierarchies  in  social  organizations.  He  notes 
that  sub-systems  exist  only  as  entities  relative  to  their  positions  in  the  hierarchy.  He  proposes 
the  word  “holon”  to  describe  the  hybrid  nature  of  individual  organizations  within  larger 
organizations/systems.  Holons  are  unique  and  self-contained  wholes  to  their  subordinated 
parts.  But  at  the  same  time,  they  are  also  dependent  parts  of  the  larger  hierarchy  (or  “holarchy,” 
as  Koestler  termed  structures  consisting  of  them).  He  views  holons  as  autonomous,  self-reliant 
units  which  have  their  own  independence,  and  which  cope  with  contingencies  without  asking 
higher  authorities  for  instructions.  Yet,  they  remain  subordinate  ultimately,  subject  to  control 
from  higher  authorities.  The  term  seems  somewhat  analogous  to  edge  in  the  conceptualization 
of  Edge  organizations  (Alberts  &  Hayes,  2003).  Such  concepts  of  unit  knowledge, 
empowerment  and  relative  autonomy  within  organizational  structures  are  key  to  our  design  of 
various  organizations  for  experimentation. 


Organization  Theory 

Classic  organization  theory  holds  that  organizational  structures  must  change  in  response 
to  contingencies  of  size,  technology,  environment  and  other  factors.  Indeed,  it  is  accepted 
widely  that,  when  faced  with  uncertainty  (a  situation  with  less  information  than  is  needed),  the 
appropriate  management  response  should  be  either  to  redesign  the  organization  for  the  task  at 
hand  or  to  improve  information  flows  and  processing  (Galbraith,  1973).  Van  Creveld  (1985) 
applies  this  same  principle  to  command  and  control  of  combat  elements  in  war.  He  argues  that 
the  command  structure  must  either  create  a  greater  demand  for  information  (vertically, 
horizontally,  or  both)  and  increase  the  size  and  complexity  of  the  directing  organization,  or  it 
must  enable  the  local  forces  to  deal  semi-independently  with  the  situation.  His  central  theme  is 
that  decentralized  control  is  the  superior  method  of  dealing  with  uncertainty,  whether  with  the 
task  at  hand  or  with  transformation  of  the  organization  itself.  Research  by  Van  de  Ven  and 
Delbecq  (1986)  has  shown  further  that  as  complexity  and  uncertainty  increase,  hierarchical 
management  control  and  vertical  communication  strategies  are  considered  inferior  to  less 
formal  organizations  with  horizontal  communication  channels. 

Another  classical  concept  of  organizational  theory  is  Ashby’s  Law  of  Requisite  Variety 
(Ashby,  1960).  This  states  loosely  that,  in  order  to  cope  with  the  variety  of  challenges  imposed 
by  it,  the  internal  capabilities  of  a  system  must  be  as  diverse  as  those  required  by  its 
environment.  Organizational  evolution  and  survival  are  dependent  upon  requisite  variety, 
particularly  in  environmental  contexts  that  are  dynamic  and  unpredictable.  This  suggests,  too, 
that  the  organization’s  structure  and  control  strategy  must  be  matched  to  its  environment  to 
enhance  performance.  Open  and  flexible  management  styles  and  processes  are  required  often 
for  dynamic  market  and  technological  conditions.  Further,  research  by  Burrell  and  Morgan 
(Morgan,  1997)  indicates  that  any  incongruence  among  management  processes  and  the 
organization’s  environment  tend  to  reduce  organizational  effectiveness. 

What  the  cumulative  research  appears  to  support  is  that,  for  large,  complex  hierarchies 
such  as  the  Department  of  Defense — which  operate  in  today’s  environment  of  program 
complexity,  evolving  requirements,  and  rapidly  changing  technology — decentralized  control  and 
empowerment  should  be  an  organizational  strength.  Notwithstanding  such  cumulative  research, 
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however,  organizational  hierarchies  persist  (Leavitt,  2004).  Indeed,  for  DoD  acquisition  in 
particular,  the  command  structure  has  remained  relatively  stable  since  the  late  1980s.  Although 
the  current  command  structure  is  arguably  flatter  and  more  streamlined  now  than  it  was  in  the 
Seventies  and  before,  it  remains  fundamentally  hierarchical,  centralized  and  rule-driven.  Only 
through  the  major  reform  initiatives  of  the  1980s  and  1990s  did  the  acquisition  organization’s 
“chain  of  command”  become  as  streamlined  as  it  now  is  (Packard  Commission,  1986). 

Computational  Experimentation 

Drawing  heavily  from  Nissen  and  Buettner  (2004),  we  assert  that  throughout  the  era  of 
modern  science,  a  chasm  has  persisted  between  laboratory  and  field  research.  On  one  side,  the 
laboratory  provides  unparalleled  opportunity  for  controlled  experimentation.  Through 
experimentation,  the  researcher  can  manipulate  only  a  few  variables  of  interest  at  a  time  and 
can  minimize  the  confounding  associated  with  the  myriad  factors  affecting  complex  systems  and 
processes  in  the  field  (Box  et  al.,  1978;  Johnson  &  Wichern,  1992).  However,  limitations  of 
laboratory  experimentation  are  known  well  (Campbell  &  Stanley,  1973)  and  are  particularly 
severe  in  the  domain  of  acquisition.  In  acquisition  experimentation,  such  limitations  center  on 
problems  with  external  validity.  Laboratory  conditions  can  seldom  replicate  the  complexity, 
scope  and  scale  of  the  physical  organizations  and  systems  of  interest  for  research.  Experiments 
also  include  problems  with  generalizability.  Many  experiments  utilize  samples  of  convenience 
(esp.  university  students)  instead  of  working  professionals.  This  practice  calls  into  question  how 
closely  the  associated  experimental  results  are  representative  of  acquisition  behavior  in 
operational  organizations.  These  same  concerns  pertain  also  to  analytical  methods  (e.g., 
mathematical  analysis,  optimization;  see  Chiang,  1984;  Lapin,  1985).  Most  such  methods  use 
theoretical  concepts  as  variables,  not  operationalized  constructs.  And,  of  course,  analytical 
models  do  not  involve  real  people,  systems  and  organizations. 

On  the  other  side,  field  research  provides  unparalleled  opportunity  for  realism  (Denzin  &, 
Lincoln,  1994).  The  researcher  in  the  field  can  study  full-scale  artifacts  in  operational 
environments  (Yin,  1994)  and  can  minimize  the  abstraction  away  from  working  people,  systems 
and  organizations  (Glaser  &  Strauss,  1967).  However,  limitations  of  field  research  are  also 
known  well  (Campbell  &  Stanley,  1973)  and  are  particularly  severe  in  the  acquisition  domain.  In 
acquisition  field  research,  such  limitations  center  on  problems  with  internal  validity.  Field 
research  affords  little  opportunity  for  controlled  experimentation  (cf.  Cook  &  Campbell,  1979). 
Also,  confounding  data  often  results  from  the  myriad  influences  on  complex  systems  and 
organizations  that  cannot  be  isolated  in  the  field.  This  diversity  makes  it  difficult  to  identify  and 
trace  the  causes  of  differential  behaviors — better  as  well  as  worse — in  acquisition. 

As  implied  by  the  name,  computational  experiments  are  conducted  via  computer 
simulation.  As  such,  they  offer  all  of  the  cost  and  time  advantages  of  computational  analysis. 

But,  computational  experiments  go  beyond  most  simulations.  Rigorous  experimental  designs 
are  employed  to  capture  the  benefits  of  laboratory  experimentation.  The  variables  affecting 
physical  systems  and  organizations  in  the  field  can  be  isolated  and  examined  under  controlled 
conditions.  This  type  of  analysis  also  addresses  the  internal  validity  and  confounding  limitations 
of  field  research.  Yet,  computational  experiments  can  be  conducted  at  a  fraction  of  the  cost  and 
time  required  to  set  up  and  run  experiments  with  human  subjects  in  the  laboratory.  Further, 
through  external  validation,  computational  models  can  demonstrate  fidelity  emulation  of  the  key 
qualitative  and  quantitative  behaviors  of  the  physical  systems  and  organizations  they  represent. 
This  ability  addresses  the  problems  with  external  validity  and  generalizability  noted  above. 
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It  is  important  to  note:  computational  modeling  and  simulation  are  not  new  techniques  for 
the  study  of  acquisition.  For  instance,  a  major  DoD  initiative  called  simulation-based  acquisition 
has  sought  to  educate  the  workforce  about  modeling  and  simulation  (DSMC,  1998).  And  DoD 
policy  has  called  for  extensive  use  of  modeling  and  simulation  techniques  in  program  planning 
and  execution  (Gansler,  1998).  But,  simulation-based  acquisition  has  suffered  to  date  from 
problems  with  internal  and  external  validity  alike.  Such  problems  are  not  inherent  to  simulation 
methods  or  tools  per  se.  Rather,  they  stem  from  models  lacking  theoretically  rooted  behaviors, 
externally  validated  results,  and  experimental  controls.  Our  approach  to  computational 
experimentation  obviates  such  problems  deliberately. 

Figure  4  illustrates  the  essential  elements  of  computational  experimentation  as  a 
research  method.  The  top  of  the  figure  includes  a  shape  to  depict  the  bridge  metaphor 
associated  with  this  method,  as  it  spans  a  wide  gap  between  laboratory  and  field  methods. 

From  the  left  side  of  this  “bridge,”  two  arrows  represent  inputs  to  describe  the  behaviors  of 
computational  models.  Organization  theory,  which  is  predicated  upon  many  thousands  of 
studies  over  the  last  half  century,  provides  the  basis  for  most  such  behaviors.  Behaviors 
pertaining  to  organizational  factors  such  as  centralization,  division  of  labor,  task 
interdependence,  function,  coordination,  formalization,  technology  and  information  processing 
from  organization  theory  are  captured  well.  Where  extant  theory  does  not  address  well  a 
behavior  of  interest  (e.g.,  knowledge  flows),  ethnographic  and  like  immersive  field  studies 
(Bernard,  1998)  are  conducted  to  understand  the  associated  organizational  behaviors.  Because 
organization  theory  is  general  and  not  based  on  any  single  organization,  the  associated 
behaviors  have  broad  applicability  across  organizations  in  practice.  This  provides  for  the 
generalizability  attainable  through  the  method  of  computational  experimentation. 
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Figure  4.  Bridge  Method  (adapted  from  Nissen  &  Buettner,  2004) 
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From  the  bottom  of  the  “bridge,”  an  arrow  represents  the  use  of  computer  models  to 
represent  organizations  and  to  emulate  their  key  behaviors.  Some  variety  exists  in  terms  of 
specific  implementations,  but  most  computer  models  adhere  to  standards,  norms  and 
conventions  associated  with  the  COT  field.  The  central  goal  is  to  develop  computer  models  that 
emulate  the  key  behaviors  of  organizations  and  to  use  such  models  to  examine  alternate 
methods  of  organization  and  coordination.  As  such,  COT  shares  with  acquisition  a  focus  on 
many  factors  of  importance. 

From  the  right  side  of  the  “bridge”  in  the  figure,  one  arrow  represents  a  requirement  in 
our  approach  for  model  validation.  Through  validation,  the  organizational  behaviors  emulated  by 
computer  models  are  examined  and  compared  with  those  of  operational  organizations  in  the 
field.  We  view  this  comparison  as  essential,  for  it  provides  confidence  that  the  behaviors 
emulated  by  the  computer  model  have  sufficient  fidelity  to  mirror  faithfully  the  behaviors  of  the 
operational  organizations  they  represent.  This  provides  for  the  external  validity  attainable 
through  computational  experimentation.  It  is  important  to  note,  not  all  COT  models  are 
subjected  to  such  validation.  Many  researchers  use  computational  models  to  conduct  theorem¬ 
proving  studies,  which  are  valuable  in  their  own  right  to  demonstrate  various  aspects  of 
organization  theory.  But  without  validation,  researchers  have  difficulty  making  claims  that  such 
theory  mirrors  the  behavior  of  organizations  in  the  field.  Hence,  validation  represents  an 
important  characteristic  of  distinguishing  computational  experimentation  (as  the  research 
method  described  specifically  in  this  article)  from  COT  in  general. 

Finally,  from  the  top  of  the  “bridge,”  an  arrow  represents  the  use  of  experimental  controls 
in  research.  Following  the  same,  rich  set  of  experimental  designs  available  to  laboratory 
researchers  (e.g.,  full-factorial,  Latin  Squares,  blocking  with  replication),  computational 
experimentation  as  a  research  method  can  be  used  to  control  myriad  factors  and  to  manipulate 
just  one  or  a  few  variables  at  a  time  (e.g.,  searching  for  causality  relations).  Further,  the  same 
experimental  design  and  setup  can  be  replicated  any  number  of  times — for  instance,  using 
Monte  Carlo  techniques  or  other  computational  approaches  to  introduce  variation.  This  provides 
for  the  internal  validity  attainable  through  computational  experimentation.  The  combination  of 
these  “bridge”  inputs — organization  theory  and  ethnography,  computer  models,  validation  and 
control — allows  the  method  of  computational  experimentation  to  be  understood  in  terms  of,  and 
to  indeed  inherit,  the  various  properties  of  its  constituent  elements. 

Figure  4  illustrates  also  the  bridging  nature  of  computational  experimentation  as  a 
research  method.  On  the  left  side,  we  depict  analytical  and  laboratory  methods  and  we 
summarize  their  key  advantages  (e.g.,  low-cost  &  fast  studies,  good  experimental  control  & 
internal  validity)  and  disadvantages  (e.g.,  poor  external  validity  &  generalizability).  On  the  right 
side,  we  depict  field  methods  in  similar  fashion  to  summarize  their  key  advantages  (e.g.,  good 
external  validity  and  generalizability)  and  disadvantages  (e.g.,  high  cost  &  time  consuming,  poor 
experimental  control  &  internal  validity).  Notice  from  their  relative  advantages  and 
disadvantages  how  the  two  classes  of  research  methods  complement  one  another.  Field 
methods  are  strong  in  the  areas  where  analytical  and  laboratory  methods  are  weak,  and  vice 
versa.  As  an  alternate  research  method,  computational  experimentation  mitigates  weakness  of 
both  classes.  For  instance,  it  enables  good  experimental  control  and  internal  validity  as  in 
laboratory  methods.  It  also  promotes  good  generalizability  and  external  validity  as  in  field 
methods. 

Nonetheless,  every  research  method  is  flawed  in  some  respects.  In  our  present  case, 
when  used  in  isolation,  computational  experimentation  is  not  as  good  as  either  method  at  its 
best.  For  instance,  because  computational  experimentation  uses  computer  models  of  people  in 
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organizations  instead  of  real  people,  it  is  weaker  in  this  respect  than  laboratory  experimentation 
is.  This  same  use  of  computer  models  instead  of  real  people  also  makes  computational 
experimentation  weaker  than  field  methods  are.  This  is  why  we  describe  computational 
experimentation  as  a  bridge  method :  it  bridges  the  chasm  between  experimental  and  field 
research  methods;  yet,  it  serves  best  to  complement,  not  to  replace,  such  methods. 

RESEARCH  DESIGN 

This  discussion  of  the  research  design  is  organized  into  three  parts:  1)  agent-based 
modeling  environment,  2)  computational  acquisition  organization  model,  and  3)  experimental 
design. 


Agent-Based  Modeling  Environment 

In  this  section,  we  build  upon  current  advances  in  VDT  research  to  describe  the  agent- 
based  modeling  environment  used  here  for  computational  experimentation.  Drawing  heavily 
from  Nissen  and  Levitt  (2004),  we  first  summarize  the  stream  of  research  associated  with  VDT 
and  then  describe  its  modeling  environment. 

Virtual  Design  Team  Research 

The  VDT  Research  Program  (VDT,  2004)  reflects  the  planned  accumulation  of 
collaborative  research  over  two  decades  to  develop  rich  theory-based  models  of  organizational 
processes.  Using  an  agent-based  representation  (Cohen,  1992;  Kunz  et  al. ,  1998),  micro-level 
organizational  behaviors  have  been  researched  and  formalized  to  reflect  well-accepted 
organization  theory  (Levitt  et  al.,  1999).  Extensive  empirical  validation  projects  (e.g., 
Christiansen,  1993;  Thomsen,  1998)  have  demonstrated  representational  fidelity  and  have 
shown  how  the  emulated  behaviors  of  VDT  computational  models  correspond  closely  with  a 
diversity  of  enterprise  processes  in  practice. 

The  VDT  research  program  continues  with  the  goal  of  developing  new  micro¬ 
organization  theory  and  of  embedding  it  in  software  tools  that  can  be  used  to  design 
organizations  in  the  same  way  that  engineers  design  bridges,  semiconductors  or  airplanes: 
through  computational  modeling,  analysis  and  evaluation  of  multiple,  alternate  prototype 
systems.  Clearly,  this  represents  a  significant  challenge  in  the  domain  of  organizations.  Micro¬ 
theory  and  analysis  tools  for  designing  bridges  and  airplanes  rest  on  well-understood  principles 
of  physics  (e.g.,  involving  continuous  numerical  variables,  describing  materials  whose  properties 
are  relatively  easy  to  measure  and  calibrate),  and  analysis  of  such  physical  systems  yields 
easily  to  differential  equations  and  precise  numerical  computing. 

In  contrast,  theories  describing  the  behavior  of  organizations  are  characterized  by 
nominal  and  ordinal  variables,  with  poor  measurement  reproducibility,  and  verbal  descriptions 
reflecting  significant  ambiguity.  Unlike  the  mathematically  representable  and  analyzable  micro¬ 
behaviors  of  physical  systems,  the  dynamics  of  organizations  are  influenced  by  a  variety  of 
social,  technical  and  cultural  factors,  are  difficult  to  verify  experimentally,  and  are  not  as 
amenable  to  numerical  representation,  mathematical  analysis  or  precise  measurement. 
Moreover,  quite  distinct  from  physical  systems,  people  and  social  interactions — not  molecules 
and  physical  forces — drive  the  behavior  of  organizations.  Hence,  such  behaviors  are 
fundamentally  non-deterministic  and  difficult  to  predict  at  the  individual  level.  Thus,  people, 
organizations  and  business  processes  are  qualitatively  different  from  bridges,  semiconductors 
and  airplanes.  And  it  is  irrational  to  expect  the  former  to  ever  be  as  understandable,  analyzable 
or  predictable  as  the  latter.  This  represents  a  fundamental  limitation  of  the  approach. 


ACQUISITION  RESEARCH:  THE  FOUNDATION  FOR  INNOVATION 


-90- 


Within  the  constraints  of  this  limitation,  however,  we  can  still  take  great  strides  beyond 
relying  upon  informal  and  ambiguous,  natural-language  textual  description  of  organizational 
behavior  (e.g.,  the  bulk  of  extant  theory).  For  instance,  the  domain  of  organization  theory  is 
imbued  with  a  rich,  time-tested  collection  of  micro-theories  that  lend  themselves  to  qualitative 
representation  and  analysis.  Examples  include  Galbraith's  (1977)  information-processing 
abstraction,  March  and  Simon’s  (1958)  bounded  rationality  assumption,  and  Thompson’s  (1967) 
task-interdependence  contingencies.  Drawing  from  this  theory  base,  we  employ  symbolic  (i.e., 
non-numeric)  representation  and  reasoning  techniques  from  established  research  on  artificial 
intelligence  to  develop  computational  models  of  theoretical  phenomena.  Once  formalized 
through  a  computational  model,  the  symbolic  representation  is  “executable,”  meaning  it  can 
emulate  the  dynamics  of  organizational  behaviors. 

Even  though  the  representation  is  qualitative  (e.g.,  lacking  the  precision  offered  by 
numerical  models),  through  commitment  to  computational  modeling,  it  becomes  semi-formal 
(e.g.,  different  people  viewing  the  model  can  agree  on  what  it  describes),  reliable  (e.g.,  the 
same  sets  of  organizational  conditions  and  environmental  factors  generate  the  same  sets  of 
behaviors),  and  explicit  (e.g.,  much  ambiguity  inherent  in  natural  language  is  obviated). 
Particularly  when  used  in  conjunction  with  the  descriptive  natural  language  theory  of  our  extant 
literature,  this  represents  a  substantial  advance.  Further,  once  a  model  has  been  validated  to 
emulate  accurately  the  qualitative  behaviors  of  the  field  organization  it  represents,  it  can  be 
used  to  examine  a  multitude  of  cases  (e.g.,  many  more  and  diverse  than  observable  in  practice) 
under  controlled  conditions  (e.g.,  repeating  the  same  events  multiple  times,  manipulating  only 
one  or  a  few  variables  at  a  time  through  repeated  trials,  stopping  the  action  for  interpretation). 
These  features  alone  offer  great  promise  in  terms  of  theory  development  and  testing. 

Additionally,  although  organizations  are  inherently  less  understandable,  analyzable  and 
predictable  than  physical  systems  are,  and  the  behavior  of  people  is  non-deterministic  and 
difficult  to  model  at  the  individual  level,  it  is  known  well  that  individual  differences  tend  to 
average  out  when  aggregated  cross-sectionally  and/or  longitudinally.  Thus,  when  modeling 
aggregations  of  people  in  the  organizational  context  (e.g.,  work  groups,  departments,  firms), 
one  can  augment,  with  certain  aspects  of  numerical  representation,  the  kind  of  symbolic  model 
from  above.  For  instance,  the  distribution  of  skill  levels  in  an  organization  can  be 
approximated — in  aggregate — by  a  Bell  Curve;  the  probability  of  a  given  task  incurring 
exceptions  and  requiring  rework  can  be  specified — organization  wide — by  a  distribution;  and  the 
unpredictable  attention  of  a  worker  to  any  particular  activity  or  event  (e.g.,  new  work  task, 
communication,  request  for  assistance)  can  be  modeled — stochastically — to  approximate 
collective  behavior.  As  another  instance,  specific  organizational  behaviors  can  be  simulated 
hundreds  of  times — such  as  through  Monte  Carlo  techniques — to  gain  insight  into  which  results 
are  common  and  expected  versus  those  that  are  rare  and  exceptional. 

Of  course,  applying  numerical  simulation  techniques  to  organizations  is  nothing  new 
(e.g.,  see  Law  &  Kelton,  1991).  But  this  approach  enables  us  to  integrate  the  kinds  of  dynamic, 
qualitative  behaviors  emulated  by  symbolic  models  with  quantitative  aggregate  dynamics 
generated  through  discrete-event  simulation.  It  is  through  such  integration  of  qualitative  and 
quantitative  models — bolstered  by  strong  reliance  upon  well-established  theory  and  commitment 
to  empirical  validation — that  our  approach  diverges  most  from  extant  research  methods  and 
offers  new  insight  into  the  dynamics  of  organizational  behavior. 
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VDT  Modeling  Environment 

Here  we  provide  a  brief  overview  of  the  VDT  modeling  environment.  The  development 
and  evolution  of  VDT  has  been  described  in  considerable  detail  elsewhere  (e.g.,  Cohen,  1992; 
Christiansen,  1993;  Jin  &  Levitt,  1996;  Thomsen,  1998;  Kunz  et  al. ,  1998;  Levitt  et  al.,  1999; 
Nogueira,  2000;  VDT,  2004),  so  we  do  not  repeat  such  discussion  here.  The  VDT  modeling 
environment  has  been  developed  directly  from  Galbraith’s  information-processing  view  of 
organizations.  This  information-processing  view  has  two  key  implications  (Jin  &  Levitt,  1996). 
The  first  is  ontological:  we  model  knowledge  work  through  interactions  of  tasks  to  be  performed, 
actors  communicating  with  one  another  and  performing  tasks,  and  an  organization  structure  that 
defines  actors’  roles  and  that  constrains  their  behaviors.  In  essence,  this  amounts  to  overlaying 
the  task  structure  on  the  organization  structure  and  to  developing  computational  agents  with 
various  capabilities  to  emulate  the  behaviors  of  organizational  actors  performing  work. 

Figure  5  illustrates  this  view  of  tasks,  actors  and  organization  structure.  As  suggested  by 
the  figure,  we  model  the  organization  structure  as  a  network  of  reporting  relations  which  can 
capture  micro-behaviors  such  as  managerial  attention,  span  of  control  and  empowerment.  We 
represent  the  task  structure  as  a  separate  network  of  activities,  which  can  capture 
organizational  attributes  such  as  expected  duration,  complexity  and  required  skills.  Within  the 
organization  structure,  we  further  model  various  roles  (e.g.,  marketing  analyst,  design  engineer, 
manager),  which  can  capture  organizational  attributes  such  as  skills  possessed,  level  of 
experience  and  task  familiarity.  Within  the  task  structure,  we  further  model  various  sequencing 
constraints,  interdependencies  and  quality/rework  loops — which  can  capture  considerable 
variety  in  terms  of  how  knowledge  work  is  organized  and  performed. 

As  suggested  also  by  the  figure,  each  actor  within  the  intertwined  organization  and  task 
structures  has  a  queue  of  information  tasks  to  be  performed  (e.g.,  assigned  work  activities, 
messages  from  other  actors,  meetings  to  attend)  and  a  queue  of  information  outputs  (e.g., 
completed  work  products,  communications  to  other  actors,  requests  for  assistance).  Each  actor 
also  processes  such  tasks  according  to  how  well  the  actor’s  skill  set  matches  those  required  for 
a  given  activity,  the  relative  priority  of  the  task,  the  actor’s  work  backlog  (i.e.,  queue  length),  and 
how  many  interruptions  divert  the  actor’s  attention  from  the  task  at  hand.  Collective  task 
performance  is  constrained  further  by  the  number  of  individual  actors  assigned  to  each  task,  the 
magnitude  of  the  task,  and  both  scheduled  (e.g.,  work  breaks,  ends  of  shifts,  weekends  and 
holidays)  and  unscheduled  (e.g.,  awaiting  managerial  decisions,  awaiting  work  or  information 
inputs  from  others,  performing  rework)  downtime. 
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Figure  5.  VDT  Information  Processing  View  of  Knowledge  Work  (adapted  from  Nissen  &  Levitt,  2004) 

The  second  implication  is  computational:  both  primary  work  (e.g.,  planning,  design, 
management)  and  coordination  work  (e.g.,  group  tasks,  meetings,  joint  problem  solving)  are 
modeled  in  terms  of  work  volume.  This  construct  is  used  to  represent  a  unit  of  work  (e.g., 
associated  with  a  task,  a  meeting,  a  communication)  within  the  task  structure.  In  addition  to 
symbolic  execution  of  VDT  models  (e.g.,  qualitatively  assessing  skill  mismatches,  task- 
concurrency  difficulties,  decentralization  effects)  through  micro-behaviors  derived  from 
organization  theory,  the  discrete-event  simulation  engine  enables  (virtual)  process  performance 
to  be  assessed  (e.g.,  quantitatively  projecting  task  duration,  cost,  rework,  process  quality). 

Clearly,  quantitative  simulation  places  additional  burden  on  the  modeler  in  terms  of 
validating  the  representation  of  a  knowledge-work  process,  which  generally  requires  fieldwork  to 
study  an  organization  in  action.  The  VDT  modeling  environment  benefits  from  extensive 
fieldwork  in  many  diverse  enterprise  domains  (e.g.,  power-plant  construction  and  offshore 
drilling,  see  Christiansen,  1993;  aerospace,  see  Thomsen,  1998;  software  development,  see 
Nogueira,  2000;  healthcare,  see  Cheng  &  Levitt,  2001;  others).  Through  the  process  of 
“backcasting” — predicting  known  organizational  outcomes  using  only  information  that  was 
available  at  the  beginning  of  a  project — VDT  models  of  operational  enterprises  in  practice  have 
demonstrated  dozens  of  times  that  emulated  organizational  behaviors  and  results  correspond 
qualitatively  and  quantitatively  to  their  actual  counterparts  in  the  field  (Kunz  et  al.,  1998). 

Viewing  VDT  as  a  validated  model  of  project-oriented  knowledge  work,  researchers 
have  begun  to  use  this  dynamic  modeling  environment  as  a  “virtual  organizational  testbench”  to 
explore  a  variety  of  organizational  questions,  such  as  effects  of  distance  on  performance  (Wong 
&  Burton,  2000)  or  to  replicate  classic  empirical  findings  (Carroll  &  Burton,  2000).  Thus,  the  VDT 
modeling  environment  has  been  validated  repeatedly  and  longitudinally  as  representative  of 
both  organization  theory  and  enterprises  in  practice.  This  gives  us  considerable  confidence  in  its 
results.  Moreover,  VDT  is  designed  specifically  to  model  the  kinds  of  knowledge  work  and 
information-processing  tasks  that  comprise  the  bulk  of  acquisition  processes. 
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Computational  Acquisition  Organizational  Model 

In  our  experimental  efforts,  we  use  the  VDT  modeling  environment  to  represent  work 
associated  with  a  three-tier  acquisition  organization.  This  follows  our  discussion  above,  and  it  is 
representative  of  many  DoD  service-level  environments  today  (e.g.,  where  several  project 
offices  report  into  one  program  executive  “Portfolio  Manager”  and  then  up  to  a  Component 
Acquisition  Executive,  and  often  into  yet  another  level  of  decision-making).  VDT  is  capable  of 
modeling  large,  complex,  operational  organizations  in  great  detail;  it  has  been  demonstrated 
repeatedly  to  emulate  well  the  associated  behaviors  of  organizations  in  the  field.  But,  using  a 
high-level  model  as  such  helps  us  to  maintain  the  focus  of  this  expository  article  on  techniques 
of  VDT  modeling  and  computational  experimentation  (which  represents  one  of  our  primary 
contributions),  and  not  to  get  lost  in  the  details  of  the  organization  itself.  We  first  describe  the 
VDT  representation  and  then  illustrate  how  a  full-factorial  computational  experiment  can  be 
performed  upon  it. 

VDT  Acquisition  Model 

Figure  6  presents  a  screenshot  of  the  VDT  acquisition  program  model.  The  model  is 
comprised  of  five  developmental  system  acquisition  projects  (i.e.,  denoted  as  lightly  colored 
boxes).  Both  concurrent  and  sequential  projects/tasks  are  depicted  in  the  model,  and 
interdependencies  are  represented  among  them.  The  model  depicts  a  simple  and  abbreviated 
series  of  coordinated  research  and  development  efforts  which  are  aligned  to  deliver  an 
Advanced  Strike  capability  integrated  into  a  mobile  platform  and,  subsequently,  are  enhanced 
with  an  evolutionary  block  of  capability.  It  is  but  a  representative  subset  of  what  could  be  a 
larger,  more  complex,  and  more  detailed  representation  of  such  a  program. 
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Figure  6.  VDT  Acquisition  Model  Screenshot 

The  coordination  links  (i.e.,  denoted  by  light  dashed  lines)  connecting  the  coordinated 
tasks  or  projects  denote  reciprocal  task  interdependencies  (Thompson,  1967),  which  suggest 
they  must  be  coordinated  closely  in  both  planning  and  execution.  For  example,  integration  of 
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strike  capabilities  into  a  mobile  platform  requires  coordination  among  engineers  for  interface 
and  configuration  control  of  hardware,  software,  and  other  factors.  VDT  emulates  the  added 
coordination  effort  associated  with  such  reciprocal  task  interdependencies.  The  rework  links 
(i.e. ,  denoted  by  dark  dashed  lines)  connecting  tasks  from  different  mission  phases  denote 
sequential  task  dependencies,  which  suggest  the  predecessor  activities  must  be  accomplished 
effectively  in  order  for  the  successors  to  perform  well.  Strike  and  Mobility  Enhancements,  for 
instance,  depend  heavily  upon  success  of  the  Initial  Strike  and  Mobility  Platform  efforts.  To  the 
extent  that  such  predecessor  work  is  not  completed  or  not  accomplished  effectively,  certain 
aspects  may  have  to  be  reanalyzed  to  correct  any  major  deficiencies. 

The  people  icons  depict  organizations  and  are  arranged  in  terms  of  the  command-and- 
control,  or  decision-making,  hierarchy.  People  icons  represent  one  or  more  human  resources, 
specified  in  Full  Time  Equivalents  (FTEs),  which  have  particular  capabilities,  skill  levels  and 
roles.  Where  a  skilled  actor’s  capability  matches  that  required  for  an  acquisition  task,  the 
resource  is  likely  to  perform  it  competently  and  within  the  time  required.  If  the  actor  has  greater 
or  lesser  skill,  the  time  required  to  perform  the  task  can  be  appreciably  shorter  or  longer,  and 
the  competency  of  performance  can  be  notably  better  or  worse,  respectively.  Where  an  actor 
does  not  possess  the  required  capability  at  all,  the  task  will  be  in  jeopardy.  Such  relationships 
are  appealing  intuitively  and  reflect  well  many  organizational  behaviors. 

A  Senior  Executive  actor  sits  atop  the  acquisition  organization  model  and  has  a  Portfolio 
Manager  reporting  to  it.  Reporting  to  the  Portfolio  Manager  are  five  individual  PMOs  (01 
through  05)  with  different  roles  and  capabilities  within  them.  For  example,  the  icon  labeled  “PM 
02”  is  responsible  for  the  technological  enhancement  of  the  initial  Strike  Capability.  Notice  the 
VDT  representation  includes  a  work  task  structure  and  an  organization  structure.  The 
assignment  links  (i.e.,  delineated  by  solid  lines)  denote  which  organizational  actors  are 
responsible  for  the  various  work  tasks.  Finally,  a  dark  trapezoid  box  is  used  to  depict  recurring 
meetings  (e.g.,  coordination  meetings,  technical  reviews,  milestone  reviews)  that  must  be 
attended  by  the  actors  connected  by  links.  Meetings  consume  actors’  resources,  but  they  also 
contribute  toward  coordination. 

All  of  the  structural  elements  (e.g.,  work  tasks,  requirements  and  interdependencies, 
actor  capabilities,  skill  levels  and  roles,  organization  structure,  task  structure  and  meeting 
requirements)  of  this  VDT  model  are  developed  by  the  authors.  Such  structural  elements  would 
clearly  be  different  for  each  unique  organization  and  process  model.  VDT  also  includes  several 
dozen  environmental  variables  with  “normal”  values  determined  empirically  by  prior  field 
research.  These  include  factors  such  as  the  level  of  uncertainty  and  noise  associated  with  a 
project,  the  inherent  propensity  of  an  organization  to  make  errors,  and  relative  concern  for 
performance  quality  associated  with  actors  at  different  levels  of  organizational  hierarchy.  These 
and  other  environmental  variables  can  be  changed  where  appropriate  to  reflect  a  wide  variety  of 
different  organizations  and  contexts.  Other  factors  can  be  changed  to  reflect  different 
organizational  designs.  For  instance,  the  level  of  centralization  and  formalization  can  be  varied 
by  changing  design  variables.  The  corresponding  VDT  model  behaviors  have  been  developed 
empirically.  We  capitalize  upon  such  empirically  developed  behaviors  to  design  and  compare 
new  acquisition  organization  models  and  subject  them  to  changing  environments. 

VDT  also  includes  several  performance  variables  for  comparison.  In  addition  to  standard 
simulation  measures  such  as  project  duration  and  cost,  VDT  also  includes  measures  such  as 
levels  of  rework,  coordination  and  delay,  in  addition  to  risk  measures  keyed  to  various  attributes 
of  importance  (e.g.,  tasks  left  undone,  missed  communications,  project-level  errors).  Some  of 
these  performance  variables  are  correlated  often  with  one  another,  whereas  others  highlight 
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tradeoffs  that  must  be  made.  In  other  words,  where  a  project  is  running  behind  schedule  but  on 
budget,  a  leader  or  manager  can  decide  to  employ  more  resources.  This  often  has  the  effect  of 
increasing  the  rate  of  progress  while  also  increasing  the  rate  of  expenditure.  Other  tradeoffs 
such  as  those  between  cost  and  risk  or  schedule  and  coordination  require  balance  in  a  similar 
fashion.  It  is  important  to  note  again  the  extensive  and  longitudinal  validation  of  VDT  provides 
considerable  confidence  that  the  organizational  behaviors  emulated  by  our  computational  model 
will  reflect  well  those  of  operational  organizations  in  the  field. 


Experimental  Design 

As  is  appropriate  for  the  cumulative  accretion  of  knowledge  through  research,  this  study 
builds  upon  prior  work  using  VDT  methods  and  tools  to  examine  alternate  organizational 
designs  and  environmental  conditions.  For  instance,  Kim  and  Burton  (2002)  use  VDT  to  model 
projects  with  varying  levels  of  task  uncertainty  and  centralization,  measuring  the  effects  on  cost, 
schedule  and  risk  as  dependent  performance  variables.  They  find  a  relationship  between 
organizational  structure  and  performance.  And  they  examine  project  risk,  measuring  the 
likelihood  that  outputs  from  a  project  will  not  be  integrated  at  completion,  or  that  the  integration 
will  have  defects.  The  study  calls  attention  to  the  impact  of  centralized  control  on  organizational 
performance  in  light  of  task  uncertainty.  It  also  suggests  that  managers  should  pay  attention  to 
such  aspects  of  organizational  structure  and  should  consider  the  importance  of  project  quality  in 
addition  to  profitability  alone.  In  another  instance,  Nissen  and  Buettner  (2004)  use  VDT  to  model 
command  and  control  in  military  missions.  They  model  organizations  having  varying  levels  of 
bureaucracy,  coordination  and  knowledge,  measuring  the  effects  on  mission  duration  and  risk 
as  dependent,  performance  variables.  They  find  a  similar  relationship  between  organizational 
structure  and  task  performance  and  overall  risk,  and  they  suggest  that  organizational  leaders 
must  choose  and  balance  the  performance  measures  that  are  most  relevant  to  the  project’s 
environment  and  desired  outcomes. 

In  this  study,  we  emulate  the  behaviors  of  three  different  modeled  organizations  which 
vary  in  degrees  of  hierarchy,  centralization  and  formalization,  and  which  are  subjected  to 
different  levels  of  environmental  stress.  Briefly,  our  three  designs  of  organizations  have  the 
same  amount  of  work  volume  to  perform,  with  the  same  level  of  team  experience  and  individual 
skills  involved.  What  differs  among  them  is  their  degree  of  autonomy  and  empowerment, 
specified  by  several  VDT  constructs.  Therefore,  we  build  upon  the  kinds  of  prior  research  noted 
above,  and  we  extend  such  prior  research  to  address  the  acquisition  domain.  We  also  extend 
prior  research  through  the  greater  number  and  variety  of  organizational  design  changes  and 
degrees  of  environmental  stress  examined  in  this  study. 

Figure  7  reflects  today’s  acquisition  organization  (labeled  “Typical”)  with  high 
centralization,  formalization  and  three  layers  of  decision  hierarchy,  somewhat  like  an  ACAT  II 
program  or  set  of  projects  within  the  DoD. 
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In  contrast  to  the  typical  organization,  Figures  8  and  9  depict  two  alternate  organizations 
with  fewer  layers  of  decision  hierarchy  and  lower  centralization  and  formalization.  The  first 
organization  (labeled  “Decentralized”)  has  less  hierarchy  and  control  overhead  in  its 
management  structure.  Note  the  removal  of  the  “Senior  Executive”  position  in  the 
representation,  whose  VDT  role  of  PM  has  been  delegated  lower  to  the  Portfolio  Manager,  now 
labeled  as  “Leader.”  As  in  reality,  the  supervision  structure  in  the  VDT  model  is  an  exception¬ 
handling  hierarchy.  It  is  the  chain  of  command  for  information  and  decision  about  problems 
discovered  in  the  course  of  a  project.  Positions  of  PM  01,  02,  and  others  still  act  within  the  VDT 
simulation  as  Subteam  Leaders  who  handle  some  exceptions  and  pass  others  up  the  hierarchy 
for  resolution. 
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Figure  8.  Decentralized  Organizational  Design  and  Project  Work 

The  second  organization,  depicted  in  Figure  9  (labeled  “Holonistic”)  has  no  overhead 
management  structure  at  all.  Here,  each  PM  position  in  the  figure  remains  designated  a 
Subteam  Leader  within  the  VDT  tool.  The  various  PMs  communicate  with  one  another  directly. 
Table  1  shows  the  VDT  settings  for  organizational  parameters  to  be  tested.  Additional  modeling 
detail  on  organizational  design  parameters  is  found  in  Appendix  A. 
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Table  1.  Organizational  Design  Parameters 


Organizational  Parameter  Typical 

Decentralized 

Holonistic 

Centralization 

High 

Low 

Low 

Formalization 

High 

Low 

Low 

Matrix  Strength 

Low 

High 

High 

Hierarchy 

3  layers 

2  layers 

1  layers 

Sr-Cmd  (Sr  Exec  PM): 

1  FTE 

0 

0 

Mid-Cmd  (Port  Mgr  SL): 

1  FTE 

1 

0 

Operations  (PMOs): 

50  FTE 

50 

50 

Communication  Links 

0 

2 

5 

Info  Exchange  Prob 

0.1 

0.9 

0.9 

Application  Exp. 

Low 

Medium 

Medium 

Meetings 

More 

Less 

None 

Functional  Error  Prob 

0.1 

0.2 

0.2 

Project  Error  Prob 

0.1 

0.2 

0.2 

Rework  Links  Str 

30 

10 

10 

Team  Experience 

Medium 

Medium 

Medium 

Skill  Level/Matched 

Medium 

Medium 

Medium 

Environmental  stress  is  applied  to  our  three  organizational  designs  via  the  VDT 
constructs  requirement  complexity,  solution  complexity,  and  task  uncertainty  (appropriate  to 
environments  that  project  offices  often  face  with  technology  maturity,  interoperability 
requirements,  etc.)  as  well  as  higher  noise  (distractions)  and  increasing  functional-  and  project- 
error  probabilities.  For  the  experiment,  each  of  these  three  factors  is  specified  at  two  levels: 
routine  and  stressed,  shown  in  Table  2  below.  Hence,  a  full-factorial  design  consists  of  six  trials 
(i.e.,  three  alternate  organizational  designs  x  2  different  environmental  conditions),  which  we 
designate  according  to  the  levels  corresponding  to  a  set  of  environmental  factors.  Additional 
modeling  detail  on  environmental  parameters  is  presented  in  Appendix  B. 

Table  2.  Environmental  Parameters 


Environmental  Parameter  Routine  Stressed 


Requirement  Complexity 

Medium 

High 

Solution  Complexity 

Medium 

High 

Uncertainty 

Medium 

High 

Noise 

0.3 

0.4 

Functional  Error  Probability 

0.1  &  0.2 

0.3  &  0.4 

Project  Error  Probability 

0.1  &  0.2 

0.3  &  0.4 
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We  examine  the  dependent  variables  of  particular  interest  in  the  acquisition  domain: 
cost,  schedule  duration  and  project  risk.  We  also  make  note  of  the  maximum  position  backlog, 
rework  volume,  coordination  volume,  and  decision  wait  time,  as  these  have  implications  for 
managers  to  consider.  Schedule  is  important  to  project  managers,  and  time  is  often  viewed  as 
money  because  of  the  staff  that  must  be  paid  as  long  as  they  are  retained — whether  productive 
to  the  project  or  not.  Project  cost  is  measured  in  $K,  and  pertains  to  staffing  costs  only,  as  no 
material  costs  are  modeled  in  our  experiment.  Project  risk,  as  mentioned  above,  is  represented 
as  the  likelihood  of  an  incomplete  project  outcome,  which  relates  directly  to  project  quality. 

While  every  task  within  a  project  may  not  be  critical  to  project  quality,  more  tasks  incomplete  or 
defective  place  the  overall  project  at  risk  for  failure.  Where  lives  are  at  stake,  such  as  in  new 
pharmaceutical  compounds,  new  passenger  aircraft,  or  defense  weapon  systems  involving 
lethality  and  survivability,  overall  project  risk  may  be  a  difficult  trade  for  managers  also 
concerned  with  project  cost  and  schedule. 

EXPERIMENTAL  RESULTS 

In  this  section,  we  report  on  the  results  of  our  computational  experiment.  Summarized  in 
Table  3  is  each  of  the  six  trials  in  this  full-factorial  experiment.  The  table  includes  measures  for 
project  cost,  schedule  and  risk,  in  addition  to  other  metrics  that  can  provide  insight  into 
organizational  dynamics  (rework  volume,  coordination  volume  and  decision  wait). 

Table  3.  Experimental  Results 


Typical  Decentralized  Holonistic  Typical  Decentralized  Holonistic 
Organization  Organization  Organization  Organization  Organization  Organization 
Measure  in  Routine  in  Routine  in  Routine  Under  Stress  Under  Stress  Under  Stress 


Duration  (dys) 

556 

428 

407 

580 

604 

458 

Cost  $K 

$8,085 

$4,674 

$4,565 

$8,561 

$6,708 

$4,973 

Project  Risk 

0.41 

0.54 

0.76 

0.37 

0.55 

0.76 

Max  Backlog  (dys) 

26 

12 

12 

30 

27 

19 

Work  Volume  (dys) 

4800 

4500 

4500 

4800 

4500 

4500 

Rework  Volume  (dys) 

124 

866 

465 

401 

2747 

740 

Coordination  Volume  (dys) 

3051 

423 

742 

3205 

952 

976 

Decision  Wait 

20 

54 

0 

67 

186 

0 

Examining  these  results,  we  see  that  the  baseline  organization — the  Typical 
Organization  in  Routine  environment — completes  the  series  of  projects  in  556  days,  at  a  cost  of 
$8,085(K),  with  a  project  risk  index  of  0.41 .  While  these  are  the  three  primary  success 
measures  of  any  project,  the  VDT  simulation  provides  more  insight  in  terms  of  position  backlog 
(e.g.,  one  actor  got  26  days  behind  in  work  at  one  point  during  the  project).  The  tool  can  also 
identify  when  this  occurs  so  that  planners  can  split  tasks  or  assign  more  resources  for  specific 
tasks.  Work  volume  refers  to  the  amount  of  effort  expected  to  complete  all  project  tasks  under 
ideal  conditions  (e.g.,  no  noise,  errors  or  miscommunications).  Rework  Volume  refers  to  the 
simulated  time  needed  for  all  positions  on  a  project  to  perform  required  rework.  Coordination 
volume  is  the  cumulative  time  positions  spend  during  a  project  processing  information  requests 
from  each  other,  attending  meetings,  and  other  coordinative  tasks.  Decision  Wait  measures  the 
cumulative  time  spent  by  positions  waiting  for  decisions  to  be  made  in  a  project.  These  values 
for  our  baseline  case  provide  a  basis  for  comparison  with  results  for  alternate  organizational 
designs  and  environmental  conditions. 
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Comparing  these  results  with  those  obtained  by  the  Decentralized  and  Holonistic 
Organizations  in  the  Routine  environment,  key  differences  are  apparent.  Decentralized  and 
holonistic  organizations  fare  considerably  better  (in  terms  of  both  cost  and  schedule)  in  the 
routine  environment  than  their  more  typical  counterpart  organizational  design  does.  Program 
schedule  or  duration  is  reduced  some  23%  (from  556  to  428  days)  with  the  Decentralized 
Organization  and  27%  (from  556  to  407  days)  by  changing  organizational  structure  toward  more 
Holonistic.  Program  cost  is  reduced  similarly  by  42%  (from  $8085  (K)  to  $4674  (K))  with 
Decentralized  and  44%  (from  $8085  (K)  to  $4565  (K))  with  Holonistic  in  the  successive  design 
iterations.  However,  project  risk  increases  appreciably  in  both  alternate  organizations,  going  up 
to  54%  and  then  to  76%,  respectively,  in  decentralized  and  holonistic  designs.  Here,  we  find 
that  Decentralized  and  Holonistic  organizational  forms  offer  a  combination  of  advantages  (e.g., 
shorter  schedule  duration,  lower  cost)  and  disadvantages  (e.g.,  higher  risk)  with  respect  to  the 
Typical  acquisition  organization  in  a  routine  environment. 

Upon  examination  of  these  organizational  designs  under  stress  environments,  we  find 
the  Typical  Organization  suffers  cost  and  schedule  growth  in  the  4-5%  range  (i.e.,  580  days, 
$8561 K),  with  a  slight  decline  in  project  risk  (0.37).  Decentralized  and  Holonistic  organizations 
under  stress  perform  better  in  the  cost  realm  with  22%  ($6708K)  and  42%  ($4973K)  reductions 
compared  with  the  Typical.  The  Decentralized  design  reveals  longer  schedule  duration  (604  vs. 
580  days),  but  the  Holonistic  organization  shows  a  21%  decrease  (458  vs.  580  days).  Again, 
project  risk  climbs  in  stress  environments  to  55%  for  Decentralized  and  to  76%  for  the 
Holonistic. 

Figure  10  illustrates  graphically  the  dynamic  relationship  we  find  between  cost  and 
organizational  design.  Notice,  in  the  routine  environment,  project  cost  decreases  abruptly  with  a 
shift  from  a  Typical  to  a  Decentralized  organizational  form.  But,  negligible  additional 
improvement  accrues  to  the  Holonistic  design.  Alternatively,  in  the  stressed  environment,  the 
Decentralized  organization  performs  better  than  the  Typical  does,  and  the  Holonistic 
organization  performs  better  still.  Notice  also  how  costs  are  higher  for  every  organizational  form 
in  the  stressed  environment  than  they  are  in  the  routine  one. 
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Figure  10.  Relationship  between  Cost  and  Organizational  Design 

Figure  1 1  illustrates  the  relationship  between  risk  and  organizational  design.  Here,  we 
observe  a  monotonic  increase  in  risk  corresponding  to  progression  in  organizational  form  from 
Typical,  through  Decentralized,  to  Holonistic.  As  costs  decrease  across  these  alternate 
organizational  forms,  risk  increases  in  lock  step.  Unlike  the  cost  results,  however,  the  stressed 
environment  appears  to  exert  little  influence  in  terms  of  risk. 


Figure  11.  Relationship  between  Risk  and  Organizational  Design 

Interpreting  these  results,  the  researchers  found  that  the  less  centralized,  formalized  and 
hierarchical  organizational  designs  perform  better  in  terms  of  cost  and  schedule  than  other 
designs,  but  with  accompanying  project-quality  risk.  Interpreting  these  results  further,  in  which 
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schedule  and  cost  are  of  primary  concern  to  all  project  managers,  decentralized  control 
(especially  in  stressed  environments)  may  provide  a  more  cost-effective  approach.  Alternatively, 
where  project  risk  or  quality  is  paramount,  formalized  procedures,  vertical  information  flows,  and 
centralized  decision-making  typical  of  bureaucratic  organizational  forms  can  be  seen  as 
superior.  This  reflects  a  fundamental  tradeoff  between  performance  measures  and 
organizational  design,  as  conceptualized  generally  in  terms  of  Contingency  Theory  (Lawrence  & 
Lorsch,  1967).  And  as  Kim  and  Burton  (2002)  noted,  the  theory  is  actually  extended  by  the 
evidence  of  risk  coming  into  play  with  a  more  rapid  and  inexpensive  project  solution  afforded  by 
empowered  actors  with  relevant  information  at  their  organizational  edge. 

The  DoD,  like  sponsors  of  projects  in  the  FDA’s  pharmaceutical  arena  and  in  the  FAA’s 
commercial  aviation  arena,  is  averse  generally  to  risk  due  to  the  safety  and  survivability  aspects 
of  many  of  its  developmental  systems.  Indeed,  the  modeling  here  can  be  viewed  as 
confirmation  of  the  DoD’s  varying  levels  of  decision  hierarchy  correlating  to  estimated  program 
dollar  thresholds  (stratification  of  acquisition  categories  I  though  IV)  as  a  means  of  addressing 
cost  risk.  However,  and  just  as  important  to  illustrate,  high  levels  of  bureaucracy  place 
considerable  stress  on  acquisition  organizations  and  come  at  their  own  cost.  Are  40%  program 
cost  growth  and  25%  schedule  growth  commensurate  with  20  -  50%  program  risk  reduction? 
Might  a  commensurate  amount  of  risk  be  alleviated  through  a  less-expensive  means?  Clearly, 
tools  such  as  VDT  provide  a  new  way  of  gaining  insights  into  these  important  program 
considerations,  particularly  when  forming  organizations  for  the  management  of  weapon  system 
developments. 

CONCLUSION 

The  DoD  is  a  large,  bureaucratic,  rule-intensive  organization  that  may  not  be  suited  well 
for  its  environment.  Building  upon  prior  research  on  acquisition  centralization  and  knowledge 
dynamics,  we  employ  computational  methods  to  assess  the  behavior  and  performance  of 
different  organizational  designs  in  varying  environments.  Our  results  reinforce  Contingency 
Theory  and  suggest  particular  characteristics  of  different  acquisition  environments  that  make 
one  form  relatively  more  or  less  appropriate  than  another.  Practically,  answers  to  our  research 
questions  have  direct  and  immediate  application  to  acquisition  leaders  and  policy  makers. 
Theoretically,  we  generalize  to  broad  classes  of  organizations  and  prescribe  a  novel  set  of 
organizational  design  guides. 

In  this  study,  we  use  the  VDT  modeling  environment  to  represent  and  emulate  the 
behavior  of  an  acquisition  organization.  Although  the  Typical  acquisition  organization  modeled 
in  this  study  is  representative  of  such  organizations  in  practice,  we  do  not  claim  to  have 
experimented — even  computationally — with  an  operational  organization.  Rather,  we  experiment 
computationally  with  a  high-level  organizational  model,  illustrating  the  method,  use  and  utility  of 
our  approach  for  exposition.  We  then  conceptualize  and  model  two  alternate  acquisition 
organizations,  manipulating  key  factors  of  their  organizational  designs.  We  subject  them  to  two 
environmental  contexts,  routine  and  stressed,  comparing  their  performance  in  terms  of  cost, 
schedule  and  risk. 

In  routine  circumstances,  our  experimental  decentralized  and  holon-type  organizations 
out-perform  typical  hierarchies  in  measures  of  cost  and  schedule.  Under  high  stress  from  task 
uncertainty,  noise,  and  error  probability,  our  decentralized  and  holon-type  organizations 
completed  their  same  project  work  volume  as  well,  faster,  and  for  less  cost  than  their  centralized 
counterpart.  In  both  environments,  however,  our  less  formal  organization  structures  yield  a 
higher  project-quality  risk. 
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Our  findings  are  similar  to  those  of  other  VDT  researchers  who  find  the  relationship 
between  organizational  performance  improvements  and  increasing  project  risk  from 
decentralization  in  environments  of  uncertainty  (Kim  &  Burton,  2002)  and  worker  knowledge 
(Nissen  &  Buettner,  2004).  They  offer  an  extension  of  contingency  theory  to  include  risk  as  a 
dependent  variable  for  organizational  structures  and  project  outcomes.  Our  results  reveal  the 
same  relational  patterns  of  performance  capabilities  among  the  three  organizational  designs 
and  across  differing  stress  environments.  They  underscore  complex  interactions  between 
organizational  design  factors,  and  suggest  fundamental  tension  and  decision  tradeoffs  between 
important  performance  measures  such  as  project  cost,  schedule  and  quality/risk. 

The  results  provide  several  implications  for  managerial  practices  and  application  of 
organization  theory  regarding  the  relationships  between  organizational  structure  and 
performance.  Understanding  when  the  bureaucracy  is  relatively  beneficial  and  how  this  rigid 
organizational  form  can  negatively  influence  project  cost  (and  positively  impact  project  risk)  is 
important  for  acquisition  practitioners  today.  The  apparent  implications  are  that  adopting  a 
decentralized  structure  in  accordance  with  contingency  theory  may  not  lead  to  higher  unit 
performance,  since  it  might  instead  produce  poorer  project  quality.  But,  we  suspect  it  is 
insufficient  to  only  assume  that  more  bureaucracy  alleviates  risk  with  attendant  costs,  or  that 
managers  must  simply  choose  either  fast  and  cheap,  or  better  quality  results. 

In  the  early  1990’s,  with  a  goal  of  shortening  development  times,  reducing  cost,  and 
increasing  numbers  of  scientific  missions  flown,  NASA  adopted  a  “Faster,  Better,  Cheaper” 
approach  to  project  management.  This  management  philosophy  was  implemented  in  spite  of 
an  old  project  management  adage  that  project  managers  could  have  any  two  of  these 
performance  outcomes,  but  not  all  three  (Spear,  2000).  This  maxim  is  supported  somewhat  by 
the  findings  of  Lin  and  Carley  (1997)  regarding  decision  accuracy  in  organizations  under  time 
pressure.  After  the  several  unmanned  mission  failures,  and  ultimately  the  February  2003 
Columbia  disaster  that  claimed  the  lives  of  seven  astronauts,  an  analysis  of  NASA  failures 
blamed  a  more  risk-tolerant  culture  as  an  organizational  cause  of  the  accident. 

The  DoD,  having  large  complex  systems  with  inherent  risk  of  their  own,  is  particularly 
averse  to  risk  in  its  decision  structure  and  perhaps  even  its  organizational  culture.  With  growing 
federal  budget  deficits  and  base  re-alignments,  it  is  also  particularly  cost-constrained.  And  with 
accelerating  obsolescence  rates  of  weapon  system  technology,  the  DoD  remains  under 
considerable  pressure  to  reduce  project  schedules  as  well.  Even  with  simple  models,  we  show 
that  project  performance  can  be  examined  with  various  organizational  designs  and  under 
differing  environments.  Perhaps  for  the  first  time — or  at  least  to  an  extent  unachievable 
heretofore — we  show  how  managers  can  gain  fundamental  insights  into  the  inherent  project 
tradeoffs  in  advance  of  making  project  decisions.  The  practical  significance  should  be  apparent 
immediately. 

These  experiments  support  propositions  that  information  processing  is  a  primary 
organizational  activity  and  is  associated  with  project  cost  and  duration  (i.e.  the  more  information 
processing  a  project  requires,  the  more  costly  and  lengthy  the  project  becomes).  Certainly, 
there  is  attendant  benefit  to  the  information  processed  as  well.  However,  the  additional 
measure  of  project-quality  risk  is  critical  for  many  types  of  projects,  and  its  emerging  relationship 
from  these  studies  and  our  most  recent  work  begin  to  shape  a  new  hypothesis:  that  perhaps 
there  is  an  optimal  organizational  design  solution,  relative  to  cost,  duration  and  risk.  If 
managers  can  ascertain  early  on  the  criticality  (and  tolerable  level)  of  project-quality  risk,  they 
can  perhaps  select  along  a  continuum  the  level  of  organizational  hierarchy  and  centralization 
needed  to  control  project  outcomes.  Or,  reframing  the  question,  how  much  will  added 
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bureaucracy  cost  to  alleviate  risk?  The  key  point  is:  the  answer  will  differ — necessarily — for 
every  project.  A  one-size-fits-all  acquisition  policy  is  naive  given  such  knowledge  and  our  ability 
to  emulate  organizational  performance  as  illustrated  in  this  article. 

Building  upon  the  VDT  constructs  introduced  in  this  article,  one  day  researchers  may 
even  develop  techniques  for  design  optimization  based  on  project  objectives  (e.g.,  speed  vs. 
risk)  and  environment.  Leaders,  managers  and  researchers  may  develop  the  capability  to 
design  organizations,  work  processes  and  technologies  using  computational  techniques 
comparable  to  those  employed  for  designing  airplanes,  bridges  and  computers.  That  day  is  not 
yet  here.  But,  through  research  along  these  lines,  we  can  both  foresee  and  accelerate  its  arrival. 
Meanwhile,  the  centralized  control  that  dominates  current  acquisition  thinking  and  policy  merits 
re-examination  in  light  of  this  study.  Such  control  imposes  costs  as  well  as  accrues  benefits.  We 
know  now  how  to  measure  and  compare  them:  via  computational  experimentation. 
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APPENDIX  A:  DESIGN  OF  ORGANIZATION  MODELS 


This  appendix  provides  additional  detail  about  the  design  of  our  experimental  acquisition 
organization  models.  Table  1  above  specifies  design  parameters  and  VDT  simulation  settings 
for  all  three  of  our  organizational  designs:  Typical,  Decentralized  and  Holonistic.  We  reproduce 
the  information  below  as  Table  4  and  discuss  the  various  design  parameters. 

Table  4.  Organizational  Design  Parameters 


Organizational  Parameter  Typical  Decentralized  Holonistic 


Centralization 

High 

Low 

Low 

Formalization 

High 

Low 

Low 

Matrix  Strength 

Low 

High 

High 

Hierarchy 

3  layers 

2  layers 

1  layers 

Sr-Cmd  (Sr  Exec  PM): 

1  FTE 

0 

0 

Mid-Cmd  (Port  Mgr  SL): 

1  FTE 

1 

0 

Operations  (PMOs): 

50  FTE 

50 

50 

Communication  Links 

0 

2 

5 

Info  Exchange  Prob 

0.1 

0.9 

0.9 

Application  Exp. 

Low 

Medium 

Medium 

Meetings 

More 

Less 

None 

Functional  Error  Prob 

0.1 

0.2 

0.2 

Project  Error  Prob 

0.1 

0.2 

0.2 

Rework  Links  Str 

30 

10 

10 

Team  Experience 

Medium 

Medium 

Medium 

Skill  Level/Matched 

Medium 

Medium 

Medium 

Our  experiments  simulate  the  acquisition  efforts  of  five  small  project-management 
offices  oriented  on  an  initial  strike  capability  (such  as  one  provided  by  a  missile  or  direct-fire 
weapon),  integration  into  a  mobile  platform  (such  as  a  ground  or  air  vehicle),  and  followed  by 
block  enhancements  to  both  sets  of  capability  from  insertion  of  technology.  These  types  of 
effort  are  common  within  program  executive  office  portfolios  across  the  military  services.  To 
simulate  the  completion  of  these  projects,  we  design  three  different  organizations  with  similar 
resources,  but  varying  parameters  of  organizational  control. 

Centralization,  Formalization  and  Matrix  Strength  comprise  a  group  of  variables  that 
work  together  in  our  modeling  tool  to  characterize  levels  of  bureaucratic  organizational  control. 
Low  Centralization  settings  in  the  Decentralized  and  Holonistic  models  mean  decisions  are 
made  by  the  individual  responsible  positions.  Centralization  reflects  decision-making  in  the 
organization — either  by  senior  project  positions  or  by  “decentralized”  individuals.  With  high 
Centralization,  there  is  more  communication  required.  Formalization  measures  how  formal  the 
communication  is  in  an  organization,  with  high  Formalization  meaning  that  communication  tends 
to  occur  in  formal  meetings  (despite  the  many  informal  communication  occurring  in  any  typical 
project  office),  and  low  settings  reflecting  informal  communication  among  positions.  Matrix 
strength  characterizes  the  probability  that  workers  will  attend  to  exchanges  of  information — 
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whether  via  meetings,  communications  about  tasks,  or  noise.  It  conveys  “connectedness”  and 
can  correspond  to  geographical  collocation  of  workers.  Typical  major  program  acquisition 
organizations  have  workers  and  decision  makers  distributed  across  the  country,  and  there  is  a 
greater  need  for  meetings.  The  low  setting  for  Typical  acquisition  organizations,  however, 
reflects  high  meeting  quality  and  complements  high  Formalization.  Conversely,  high  Matrix 
Strength,  which  complements  low  Formalization,  characterizes  our  flatter,  more  Decentralized 
or  Holonistic  organizations.  In  the  “Typical”  model,  we  use  a  three-level  management  hierarchy 
with  two  different  full-time  equivalents  (FTE)  in  two  management  positions,  acting  as  portfolio 
manager  and  senior  executive.  We  reduce  to  two,  and  then  one  in  the  other  derivations.  The 
PM  and  SL  designations  beside  their  positions  are  VDT  designations,  which  connote  decision¬ 
making.  The  50  FTEs  aside — Operations  represent  project  management  office  personnel  in  five 
distinct  project  work  areas  that  are  interdependent,  and  become  more  strengthened  with 
communication  links,  used  as  such  by  the  VDT.  In  the  Typical  DoD  acquisition  organization,  we 
have  observed  it  is  common  for  individual  project  offices  within  a  PEO  to  communicate 
infrequently,  though  there  is  a  great  deal  of  vertical  communication  within  the  hierarchy, 
evidenced  in  the  model  with  Meetings.  We  also  reduce  the  number  of  meetings  in  the 
successively  flatter  organizational  designs. 

Correspondingly,  hierarchical  communication  is  also  depicted  via  a  low  setting  (0.1)  of 
Information  Exchange  Probability  for  the  Typical  Organization  and  growing  much  higher  (0.9)  in 
the  flatter  designs.  This  is  characteristic  for  a  project  involving  mostly  routine  daily  jobs 
performed  by  skilled  workers  (Typical  design).  A  higher  value  is  given  the  designs  with  more 
highly  interdependent  tasks  that  are  being  performed  by  very  busy  workers.A  low  setting  of 
Application  Experience  for  the  Typical  organization,  and  set  at  the  entire  program  level, 
describes  how  many  new  R&D  projects  the  positions  may  have  worked  on  before,  in  spite  of 
relative  individual  skill  levels  (which  are  all  presumed  as  Medium  and  Matched  to  the  work  tasks 
assigned  across  all  three  designs).  The  Decentralized  and  Holonistic  learning  organizations  are 
envisioned  as  learning  organizations,  with  the  benefit  of  some  Application  Experience  (set  at 
medium),  from  less  complicated  information  processing  and  learning. 

Five  Rework  links  connect  all  tasks  in  the  Holonistic  design,  given  the  lack  of  an 
overhead  hierarchy  as  an  integrative  function.  This  is  opposed  to  two  links  in  the  Decentralized 
and  Typical  designs.  Likewise,  Rework  Strength  designations  shift  to  reflect  success/failure 
dependency  as  higher  in  the  Decentralized  and  Holonistic,  and  lower  in  the  Typical  hierarchy, 
commensurate  with  associated  task  interdependency.  In  much  the  same  way,  the  stove-piped, 
independent  efforts  within  the  Typical  organization  hierarchy  are  represented  in  lower 
Functional  and  Project  Error  Probability  settings.  Higher  settings  for  the  flatter  organizational 
designs  convey  the  challenge  of  integration  and  alignment  they  must  face  without  an 
overarching  control  entity. 

The  total  simulated  work  task  effort  of  all  organizational  designs  is  the  same,  except  that 
layers  of  management  in  the  Typical  configuration  have  their  own  management  tasks  that  run 
about  25%  of  the  duration  of  the  project’s  planned  timeline.  The  team  experience  value  in  the 
VDT  tool  affects  the  amount  of  information  exchange  on  the  project  and  the  way  a  position’s 
information  processing  speed  is  calculated.  Team  Experience  for  the  work  effort  is  set  to 
medium  for  all  organizations,  representing  a  measure  of  how  successfully  the  team  has 
performed  related  projects. 
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APPENDIX  B:  ENVIRONMENTAL  PARAMETERS 


This  appendix  provides  additional  detail  about  the  environmental  scenarios  that  our 
experimental  acquisition  organization  models  were  subjected  to.  Table  2  above  specifies 
environmental  parameters  and  VDT  simulation  settings  for  both  of  our  environmental  conditions: 
Routine  and  Stressed.  We  reproduce  the  information  below  as  Table  5  and  discuss  the  various 
parameters. 


Table  5.  Environmental  Parameters 


Environmental  Parameter 

Routine 

Stressed 

Requirement  Complexity 

Medium 

High 

Solution  Complexity 

Medium 

High 

Uncertainty 

Medium 

High 

Noise 

0.3 

0.4 

Functional  Error  Probability 

0.1  &  0.2 

0.3  &  0.4 

Project  Error  Probability 

0.1  &  0.2 

0.3  &  0.4 

Requirement  Complexity  describes  the  degree  of  task  complexity,  which  is  relative  to  the 
total  number  of  project  requirements  that  the  task  must  satisfy.  Representative  of  current  DoD 
acquisition  environments  are  state-of-the-art  technologies  and  interoperability  requirements.  As 
such,  there  is  a  common  environment  of  at  least  a  medium  setting  within  our  VDT  tool  for 
organizations  under  even  routine  circumstances.  An  even  more  highly  optimized  design  could 
have  many  tasks  with  a  high  requirement  complexity,  and  is  appropriate  for  our  “stressed” 
settings. 

Solution  Complexity  represents  the  number  of  solutions  to  which  a  task  contributes.  The 
degree  of  complexity  reflects  the  effect  a  task  has  on  the  tasks  that  depend  on  it.  Thus,  for 
routine  circumstances,  we  use  “medium”  and  “high”  for  stressed  scenarios. 

Uncertainty  is  a  setting  regarding  the  amount  of  communication  across  links  needed  for 
a  task’s  (and  its  dependent  tasks’)  completion.  Task  uncertainty  reflects  the  effect  that  other 
tasks  can  have  on  each  other  within  the  project.  Task  Coordination  volume  and  the  number  of 
communications  increase  with  higher  uncertainty,  so  we  selected  “medium”  for  routine  and 
“high”  for  stressed  environments. 

Noise  Probability  describes  the  interruptions  or  distractions  that  detract  from  work  on 
project  tasks.  The  probability  of  noise  is  set  at  0.3  for  our  routine  scenarios  and  0.4  for  stressed. 

Functional  error  probability  is  the  probability  that  a  task  will  fail  and  require  rework. 
Functional  errors  are  localized  to  an  individual  task  and,  thus,  only  cause  rework  in  that  task. 
Functional  errors  could  be  discovered  via  self-check,  a  project-review  meeting,  or  a  supervisory 
review.  Depending  on  the  level  of  centralization  and  hierarchy  in  the  project,  an  exception  can 
be  handled  by  the  responsible  position  or  someone  up  the  hierarchy.  When  a  functional  error  is 
detected,  an  exception  is  sent  to  the  responsible  position  or  to  a  supervisor,  generating  either 
rework  of  the  task,  a  quick  fix,  or  feigned  ignorance  of  the  problem.  Project  error  probability  is 
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the  probability  that  a  task  will  fail  and  generate  rework  for  all  dependent  tasks  connected  to  it  by 
rework  links.  The  more  rework  links  there  are  in  a  project,  the  more  rework  is  generated  by  the 
exceptions  that  occur.  We  select  0.1  as  our  routine  setting  for  the  Typical  organization  and  0.2 
for  our  Decentralized  and  Holonistic  designs,  reflecting  their  decreasing  management  potential 
for  intervention.  For  stressed  scenarios,  we  use  0.3  for  our  Typical  organization  and  0.4  for  our 
Decentralized  and  Holonistic  designs. 
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